Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

cli: enable loading default profiles turnkey #943

Merged
merged 7 commits into from
Mar 29, 2021
Merged

Conversation

adam-stokes
Copy link
Contributor

@adam-stokes adam-stokes commented Mar 25, 2021

Fixes #933

Signed-off-by: Adam Stokes [email protected]

What does this PR do?

This does a little bit of refactoring of how the profiles are loaded and used. Previously, the packed files were only concerned with locating and accessing a docker-compose.yml in the profiles and services. This change allows us to put additional files within the packed binary that may be useful for running profiles and services from the cli. This also forces the cli to utilize the profiles/services after they've been written to disk in the workspace ~/.op. The files that are packed by default will always be overwritten meaning any changes to our current set of profiles/services will be the latest within the ~/.op workspace. Custom profiles/services can still be defined by either making a copy of one of the pre-existing profiles/services or creating entirely new directories containing a docker-compose.yml file.

Why is it important?

In order to have the cli experience mirror what happens in CI it is nice to be able to run the commands just as they would be run in CI, but locally. This change allows us to have the same deployments locally that would be done in CI giving us a more unified experience when writing our BDD tests.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have run the Unit tests for the CLI, and they are passing locally
  • I have run the End-2-End tests for the suite I'm working on, and they are passing locally
  • I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

  • [ ]

How to test this PR locally

To get fleet deployed:

cd cli
GO111MODULE=on OP_LOG_LEVEL=TRACE go run main.go run profile fleet

Related issues

Use cases

Screenshots

Logs

Follow-ups

@adam-stokes adam-stokes added the enhancement New feature or request label Mar 25, 2021
@adam-stokes adam-stokes requested a review from a team March 25, 2021 20:17
Signed-off-by: Adam Stokes <[email protected]>
@adam-stokes adam-stokes marked this pull request as ready for review March 25, 2021 20:26
@elasticmachine
Copy link
Contributor

elasticmachine commented Mar 25, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #943 updated

  • Start Time: 2021-03-26T20:29:51.331+0000

  • Duration: 24 min 0 sec

  • Commit: b96d41a

Test stats 🧪

Test Results
Failed 2
Passed 135
Skipped 0
Total 137

Trends 🧪

Image of Build Times

Image of Tests

Test errors 2

Expand to view the tests failures

Initializing / End-To-End Tests / ubuntu-18.04_fleet_backend_processes / Un-installing the installed centos agent – Backend Processes
    Expand to view the error details

     Step the "elastic-agent" process is in the "stopped" state on the host 
    

  • no stacktrace
Initializing / End-To-End Tests / ubuntu-18.04_fleet_fleet_mode_agent / Un-installing the installed centos agent – Fleet Mode Agent
    Expand to view the error details

     Step the file system Agent folder is empty 
    

  • no stacktrace

Steps errors 2

Expand to view the steps failures

Run functional tests for fleet:fleet_mode_agent && ~@nightly && ~debian
  • Took 8 min 33 sec . View more details on here
  • Description: .ci/scripts/functional-test.sh "fleet" "fleet_mode_agent && ~@nightly && ~debian" "8.0.0-SNAPSHOT" "8.0.0-SNAPSHOT"
Run functional tests for fleet:backend_processes && ~@nightly && ~debian
  • Took 10 min 10 sec . View more details on here
  • Description: .ci/scripts/functional-test.sh "fleet" "backend_processes && ~@nightly && ~debian" "8.0.0-SNAPSHOT" "8.0.0-SNAPSHOT"

Log output

Expand to view the last 100 lines of log output

[2021-03-26T20:52:59.986Z] Successfully enrolled the Elastic Agent.
[2021-03-26T20:53:00.246Z] time="2021-03-26T20:53:00Z" level=debug msg="Docker compose executed." cmd="[exec -T centos-systemd /elastic-agent/elastic-agent install --force --insecure --enrollment-token=Q0I1UGNIZ0JTMEJPWllxWG5kdGc6MVFCZ0ZhY2NRcXFObl8zeFRXbEIydw== --kibana-url http://kibana:5601]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/services/centos-systemd/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz245612687 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz centos_systemdContainerName:fleet_centos-systemd_elastic-agent_1 centos_systemdTag:latest stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2021-03-26T20:53:00.246Z] time="2021-03-26T20:53:00Z" level=info msg="Hostname retrieved from the Docker client" containerName=fleet_centos-systemd_elastic-agent_1 hostname=56635521bae2
[2021-03-26T20:53:00.246Z] time="2021-03-26T20:53:00Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:00.246Z] time="2021-03-26T20:53:00Z" level=warning msg="The Agent is not in the online status yet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 elapsedTime=50.875676ms hostname=56635521bae2 isAgentInStatus=false retry=1 status=online
[2021-03-26T20:53:00.824Z] time="2021-03-26T20:53:00Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:00.824Z] time="2021-03-26T20:53:00Z" level=warning msg="The Agent is not in the online status yet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 elapsedTime=599.620646ms hostname=56635521bae2 isAgentInStatus=false retry=2 status=online
[2021-03-26T20:53:01.772Z] time="2021-03-26T20:53:01Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:01.772Z] time="2021-03-26T20:53:01Z" level=warning msg="The Agent is not in the online status yet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 elapsedTime=1.615761261s hostname=56635521bae2 isAgentInStatus=false retry=3 status=online
[2021-03-26T20:53:05.067Z] time="2021-03-26T20:53:04Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:05.067Z] time="2021-03-26T20:53:04Z" level=warning msg="The Agent is not in the online status yet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 elapsedTime=4.556494821s hostname=56635521bae2 isAgentInStatus=false retry=4 status=online
[2021-03-26T20:53:07.620Z] {"level":"debug","time":"2021-03-26T20:53:07Z","message":"gathering metrics"}
[2021-03-26T20:53:10.254Z] time="2021-03-26T20:53:09Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:10.254Z] time="2021-03-26T20:53:09Z" level=warning msg="The Agent is not in the online status yet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 elapsedTime=9.774305869s hostname=56635521bae2 isAgentInStatus=false retry=5 status=online
[2021-03-26T20:53:13.620Z] time="2021-03-26T20:53:12Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:13.620Z] time="2021-03-26T20:53:12Z" level=warning msg="The Agent is not in the online status yet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 elapsedTime=12.879025391s hostname=56635521bae2 isAgentInStatus=false retry=6 status=online
[2021-03-26T20:53:17.830Z] {"level":"debug","time":"2021-03-26T20:53:17Z","message":"sent request with 0 transactions, 0 spans, 0 errors, 1 metricset"}
[2021-03-26T20:53:19.734Z] time="2021-03-26T20:53:19Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:19.734Z] time="2021-03-26T20:53:19Z" level=info msg="The Agent is in the desired status" elapsedTime=19.32231841s hostname=56635521bae2 isAgentInStatus=true retries=7 status=online
[2021-03-26T20:53:19.734Z] time="2021-03-26T20:53:19Z" level=debug msg="Integration in latest version found" name=endpoint title="Endpoint Security" version=0.18.0
[2021-03-26T20:53:23.928Z] time="2021-03-26T20:53:23Z" level=info msg="Integration added to the configuration" integration=endpoint integrationConfigurationID=ef21d4ef-6c2a-4ff4-97f1-53ec09bb984a policyID=462d28c0-8e73-11eb-93c9-69a78f490038 version=0.18.0
[2021-03-26T20:53:23.928Z] time="2021-03-26T20:53:23Z" level=debug msg="Package policy found in the configuration" package="{ef21d4ef-6c2a-4ff4-97f1-53ec09bb984a endpoint Endpoint Security 0.18.0 0xc000837600}" policyID=462d28c0-8e73-11eb-93c9-69a78f490038
[2021-03-26T20:53:26.472Z] time="2021-03-26T20:53:26Z" level=info msg="Integration deleted from the configuration" integration=endpoint packageConfigId=ef21d4ef-6c2a-4ff4-97f1-53ec09bb984a policyID=462d28c0-8e73-11eb-93c9-69a78f490038 version=0.18.0
[2021-03-26T20:53:26.472Z] time="2021-03-26T20:53:26Z" level=debug msg="Agent listed in Fleet with online status" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:26.472Z] time="2021-03-26T20:53:26Z" level=info msg="The Agent is in the desired status" elapsedTime=68.469764ms hostname=56635521bae2 isAgentInStatus=true retries=1 status=online
[2021-03-26T20:53:26.472Z] time="2021-03-26T20:53:26Z" level=info msg="The Agent is not listed in the Administration view in the Security App" elapsedTime=29.233626ms hostname=56635521bae2 retries=1
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:00.941Z","log.origin":{"file.name":"warn/warn.go","file.line":18},"message":"The Elastic Agent is currently in BETA and should not be used in production","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:00.942Z","log.origin":{"file.name":"application/application.go","file.line":66},"message":"Detecting execution mode","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"warn","@timestamp":"2021-03-26T20:53:00.942Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":101},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:00.942Z","log.origin":{"file.name":"application/application.go","file.line":91},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:00.943Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /opt/Elastic/Agent/capabilities.yml","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:01.273Z","log.logger":"composable","log.origin":{"file.name":"composable/controller.go","file.line":44},"message":"EXPERIMENTAL - Inputs with variables are currently experimental and should not be used in production","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:01.375Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.041Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:01.376Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:01.376Z","log.origin":{"file.name":"application/managed_mode.go","file.line":284},"message":"Agent is starting","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:01.376Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: /tmp/elastic-agent/elastic-agent.sock (configured: unix:///tmp/elastic-agent/elastic-agent.sock)","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"warn","@timestamp":"2021-03-26T20:53:01.477Z","log.origin":{"file.name":"application/managed_mode.go","file.line":297},"message":"failed to ack update open /opt/Elastic/Agent/data/.update-marker: no such file or directory","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:02.907Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is 12Zw6pja","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:02.908Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 3 step(s)","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:02.908Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"filebeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:02.908Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"filebeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:06.785Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:06Z: type: 'STATE': sub_type: 'STARTING' message: Application: filebeat--8.0.0-SNAPSHOT[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:06.786Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:06.786Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:07.896Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:07Z: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--8.0.0-SNAPSHOT[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to RUNNING: Running","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.042Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.662Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:09Z: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--8.0.0-SNAPSHOT[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.664Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"filebeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.664Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"filebeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.851Z","log.origin":{"file.name":"operation/operator.go","file.line":259},"message":"operation 'operation-install' skipped for filebeat.8.0.0-SNAPSHOT","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.961Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:09Z: type: 'STATE': sub_type: 'STARTING' message: Application: filebeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.962Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:09.962Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-8184a9/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:10.213Z","log.origin":{"file.name":"operation/operator.go","file.line":259},"message":"operation 'operation-install' skipped for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:10.499Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:10Z: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:10.500Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:10.804Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:10Z: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--8.0.0-SNAPSHOT[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to RUNNING: Running","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:11.108Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:11Z: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to RUNNING: Running","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.043Z] {"log.level":"info","@timestamp":"2021-03-26T20:53:11.644Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-03-26T20:53:11Z: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[ba6ce019-200c-4f93-aa5a-d57dbeec1a92]: State changed to RUNNING: Running","ecs.version":"1.6.0"}
[2021-03-26T20:53:27.305Z] time="2021-03-26T20:53:27Z" level=debug msg="Docker compose executed." cmd="[exec -T centos-systemd cat /opt/Elastic/Agent//data/elastic-agent-8184a9/logs//elastic-agent-json.log]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/services/centos-systemd/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz245612687 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz centos_systemdContainerName:fleet_centos-systemd_elastic-agent_1 centos_systemdTag:latest stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2021-03-26T20:53:29.211Z] Elastic Agent has been uninstalled.
[2021-03-26T20:53:29.211Z] time="2021-03-26T20:53:29Z" level=debug msg="Docker compose executed." cmd="[exec -T centos-systemd elastic-agent uninstall -f]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/services/centos-systemd/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz245612687 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz centos_systemdContainerName:fleet_centos-systemd_elastic-agent_1 centos_systemdTag:latest stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2021-03-26T20:53:29.211Z] time="2021-03-26T20:53:29Z" level=debug msg="Un-enrolling agent in Fleet" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92 hostname=56635521bae2
[2021-03-26T20:53:30.150Z] time="2021-03-26T20:53:29Z" level=debug msg="Fleet agent was unenrolled" agentID=ba6ce019-200c-4f93-aa5a-d57dbeec1a92
[2021-03-26T20:53:30.719Z] Stopping fleet_centos-systemd_elastic-agent_1 ... 
[2021-03-26T20:53:38.844Z] {"level":"debug","time":"2021-03-26T20:53:37Z","message":"gathering metrics"}
[2021-03-26T20:53:41.381Z] 
Stopping fleet_centos-systemd_elastic-agent_1 ... done
Removing fleet_centos-systemd_elastic-agent_1 ... 
[2021-03-26T20:53:41.382Z] 
Removing fleet_centos-systemd_elastic-agent_1 ... done
Going to remove fleet_centos-systemd_elastic-agent_1
[2021-03-26T20:53:41.382Z] time="2021-03-26T20:53:41Z" level=debug msg="Docker compose executed." cmd="[rm -fvs centos-systemd]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/services/centos-systemd/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz245612687 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz centos_systemdContainerName:fleet_centos-systemd_elastic-agent_1 centos_systemdTag:latest stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2021-03-26T20:53:41.382Z] time="2021-03-26T20:53:41Z" level=debug msg="Service removed from compose" profile=fleet service=centos-systemd
[2021-03-26T20:53:42.319Z] time="2021-03-26T20:53:42Z" level=debug msg="The token was deleted" tokenID=2bfa4558-5cad-47b8-908a-95b3ff20f31c
[2021-03-26T20:53:42.319Z] time="2021-03-26T20:53:42Z" level=info msg="Integration deleted from the configuration" integration=endpoint packageConfigId=ef21d4ef-6c2a-4ff4-97f1-53ec09bb984a policyID=462d28c0-8e73-11eb-93c9-69a78f490038 version=0.18.0
[2021-03-26T20:53:42.319Z] time="2021-03-26T20:53:42Z" level=debug msg="Destroying Fleet runtime dependencies"
[2021-03-26T20:53:43.257Z] Stopping fleet_kibana_1           ... 
[2021-03-26T20:53:43.257Z] Stopping fleet_package-registry_1 ... 
[2021-03-26T20:53:43.257Z] Stopping fleet_elasticsearch_1    ... 
[2021-03-26T20:53:44.768Z] 
Stopping fleet_kibana_1           ... done

Stopping fleet_package-registry_1 ... done

Stopping fleet_elasticsearch_1    ... done
Removing fleet_kibana_1           ... 
[2021-03-26T20:53:44.768Z] Removing fleet_package-registry_1 ... 
[2021-03-26T20:53:44.768Z] Removing fleet_elasticsearch_1    ... 
[2021-03-26T20:53:44.768Z] 
Removing fleet_package-registry_1 ... done

Removing fleet_kibana_1           ... done

Removing fleet_elasticsearch_1    ... done
Removing network fleet_default
[2021-03-26T20:53:45.341Z] time="2021-03-26T20:53:45Z" level=debug msg="Docker compose executed." cmd="[down --remove-orphans]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/.op/compose/profiles/fleet/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz245612687 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz centos_systemdContainerName:fleet_centos-systemd_elastic-agent_1 centos_systemdTag:latest stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2021-03-26T20:53:45.341Z] time="2021-03-26T20:53:45Z" level=debug msg="Elastic Agent binary was removed." installer=centos-tar-8.0.0-SNAPSHOT path=/tmp/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz245612687
[2021-03-26T20:53:45.462Z] [INFO] Stopping Filebeat Docker container
[2021-03-26T20:53:45.750Z] + docker exec -t da41ba466ef2438560eee49e19e3cc76909324f55c9d2b8d5fe8f8026be09530 chmod -R ugo+rw /output
[2021-03-26T20:53:46.009Z] + docker stop --time 30 da41ba466ef2438560eee49e19e3cc76909324f55c9d2b8d5fe8f8026be09530
[2021-03-26T20:53:46.269Z] da41ba466ef2438560eee49e19e3cc76909324f55c9d2b8d5fe8f8026be09530
[2021-03-26T20:53:46.289Z] Archiving artifacts
[2021-03-26T20:53:46.855Z] Recording test results
[2021-03-26T20:53:47.185Z] [Checks API] No suitable checks publisher found.
[2021-03-26T20:53:47.212Z] Archiving artifacts
[2021-03-26T20:53:47.255Z] Running in /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943/src/github.com/elastic/e2e-testing
[2021-03-26T20:53:47.555Z] + go clean -modcache
[2021-03-26T20:53:49.150Z] Stage "Release" skipped due to earlier failure(s)
[2021-03-26T20:53:50.529Z] Running on worker-1244230 in /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-943
[2021-03-26T20:53:50.585Z] [INFO] getVaultSecret: Getting secrets
[2021-03-26T20:53:50.688Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-26T20:53:52.804Z] + chmod 755 generate-build-data.sh
[2021-03-26T20:53:52.804Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-943/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-943/runs/7 FAILURE 1440081
[2021-03-26T20:53:52.804Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-943/runs/7/steps/?limit=10000 -o steps-info.json
[2021-03-26T20:53:54.255Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-943/runs/7/tests/?status=FAILED -o tests-errors.json
[2021-03-26T20:53:54.957Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-943/runs/7/log/ -o pipeline-log.txt

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 2
Passed 135
Skipped 0
Total 137

Genuine test errors 2

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Initializing / End-To-End Tests / ubuntu-18.04_fleet_backend_processes / Un-installing the installed centos agent – Backend Processes
  • Name: Initializing / End-To-End Tests / ubuntu-18.04_fleet_fleet_mode_agent / Un-installing the installed centos agent – Fleet Mode Agent

cli/config/config.go Outdated Show resolved Hide resolved
cli/config/config.go Outdated Show resolved Hide resolved
adam-stokes and others added 2 commits March 26, 2021 15:50
Co-authored-by: Manuel de la Peña <[email protected]>
* fix: use a more comprehensive initialisation method for configs

As go init() method is not deterministic, I found that the logger init was
not called at the right time. With change we ensure that the Init is:
1) called first
2) existing it the config was already populated

* chore: add back traces when extracting the files from the box
@adam-stokes adam-stokes merged commit ae5b9fe into master Mar 29, 2021
@adam-stokes adam-stokes deleted the gh-fix-933 branch March 29, 2021 12:15
mergify bot pushed a commit that referenced this pull request Apr 22, 2021
* cli: enable loading default profiles turnkey

Fixes #933
* update NOTICE
* Fix additional lint issues in ingest_manager_test
* Cleanup comment and trace log in GetComposeFile
* Provide better trace feedback if missing docker-compose
* Update cli/config/config.go
* chore: add back traces when extracting the files from the box (#946)
* fix: use a more comprehensive initialisation method for configs
   As go init() method is not deterministic, I found that the logger init was
   not called at the right time. With change we ensure that the Init is:
   1) called first
   2) existing it the config was already populated
* chore: add back traces when extracting the files from the box

Signed-off-by: Adam Stokes <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
(cherry picked from commit ae5b9fe)

# Conflicts:
#	cli/go.mod
#	cli/go.sum
#	e2e/_suites/fleet/ingest_manager_test.go
mergify bot pushed a commit that referenced this pull request Apr 22, 2021
* cli: enable loading default profiles turnkey

Fixes #933
* update NOTICE
* Fix additional lint issues in ingest_manager_test
* Cleanup comment and trace log in GetComposeFile
* Provide better trace feedback if missing docker-compose
* Update cli/config/config.go
* chore: add back traces when extracting the files from the box (#946)
* fix: use a more comprehensive initialisation method for configs
   As go init() method is not deterministic, I found that the logger init was
   not called at the right time. With change we ensure that the Init is:
   1) called first
   2) existing it the config was already populated
* chore: add back traces when extracting the files from the box

Signed-off-by: Adam Stokes <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
(cherry picked from commit ae5b9fe)

# Conflicts:
#	cli/go.mod
#	cli/go.sum
#	e2e/_suites/fleet/ingest_manager_test.go
mergify bot pushed a commit that referenced this pull request Apr 22, 2021
* cli: enable loading default profiles turnkey

Fixes #933
* update NOTICE
* Fix additional lint issues in ingest_manager_test
* Cleanup comment and trace log in GetComposeFile
* Provide better trace feedback if missing docker-compose
* Update cli/config/config.go
* chore: add back traces when extracting the files from the box (#946)
* fix: use a more comprehensive initialisation method for configs
   As go init() method is not deterministic, I found that the logger init was
   not called at the right time. With change we ensure that the Init is:
   1) called first
   2) existing it the config was already populated
* chore: add back traces when extracting the files from the box

Signed-off-by: Adam Stokes <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
(cherry picked from commit ae5b9fe)

# Conflicts:
#	cli/go.mod
#	cli/go.sum
#	e2e/_suites/fleet/ingest_manager_test.go
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Apr 22, 2021
* cli: enable loading default profiles turnkey

Fixes elastic#933
* update NOTICE
* Fix additional lint issues in ingest_manager_test
* Cleanup comment and trace log in GetComposeFile
* Provide better trace feedback if missing docker-compose
* Update cli/config/config.go
* chore: add back traces when extracting the files from the box (elastic#946)
* fix: use a more comprehensive initialisation method for configs
   As go init() method is not deterministic, I found that the logger init was
   not called at the right time. With change we ensure that the Init is:
   1) called first
   2) existing it the config was already populated
* chore: add back traces when extracting the files from the box

Signed-off-by: Adam Stokes <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
@mdelapenya mdelapenya mentioned this pull request Apr 22, 2021
9 tasks
mdelapenya added a commit that referenced this pull request Apr 22, 2021
* cli: enable loading default profiles turnkey (#943)

* cli: enable loading default profiles turnkey

Fixes #933
* update NOTICE
* Fix additional lint issues in ingest_manager_test
* Cleanup comment and trace log in GetComposeFile
* Provide better trace feedback if missing docker-compose
* Update cli/config/config.go
* chore: add back traces when extracting the files from the box (#946)
* fix: use a more comprehensive initialisation method for configs
   As go init() method is not deterministic, I found that the logger init was
   not called at the right time. With change we ensure that the Init is:
   1) called first
   2) existing it the config was already populated
* chore: add back traces when extracting the files from the box

Signed-off-by: Adam Stokes <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>

* Add basic check on correct service is defined for profile runs (#957)

Fixes #944

This adds a length check on the string split for verifying that the
<service/image name>:<tag> is defined when adding additional services to a
profile deployment.

Signed-off-by: Adam Stokes <[email protected]>

* Update NOTICE (#969)

Adds additional overrides to pulling in the proper licenses

Signed-off-by: Adam Stokes <[email protected]>

* Build binaries via goreleaser (#977)

This handles building for all supported architectures including running packr
for embedding the binary files.

This allows us to easily extend our release process for tagging official cli
releases, building in various package formats and publishing to different
package registries

Signed-off-by: Adam Stokes <[email protected]>

* chore: simplify release process on Jenkins (#980)

* chore: archive releases in Jenkins UI

* chore: simplify release process on Jenkins

* chore: remove garbage

* chore: support retrying fetching the goreleaser script

It will also retry in the case the release command fails

* chore: set GITHUB_TOKEN

* chore: ensure workspace is clean in the worker

* chore: add release information for goreleaser

* Remove the agent config file parameters for stand alone (#983)

Signed-off-by: Adam Stokes <[email protected]>

Co-authored-by: Manuel de la Peña <[email protected]>

* fix: run unit tests after refactor (#1067)

* chore: remove unused files after refactor

* chore: run unit tests with new layout

* fix: run unit tests on CI

* chore: include unit tests for the e2e dir

* fix: move unit tests resources for installer tests

* fix: move more test resources for unit tests

* fix: abstract path calculation from OS

Co-authored-by: Adam Stokes <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cli: unable to run basic fleet profile
5 participants